Rank Consistent Estimation: The DOP Case

The goal of an estimator is to approximate the unknown distribution of the language from its partial evidence. In this thesis, a rank consistent estimator is defined as an estimator that preserves the ranking frequencies of all the full parse trees in the treebank proved to be rank consistent with respect to the training treebank. The rank consistency property adopts Laplace’s Principle of Insufficient Reason for statistical parsing: a rank consistent estimator assigns the same probability to all trees that occur the same number of times in the training data. This thesis presents the first non-trivial DOP estimator where the treebank is not only considered as a stochastic generating system but also a sample of the stochastic process. In this thesis, the existing DOP definitions of probability and derivation of full parse trees are generalized to subtrees. Fragments in the treebank’s fragment corpus are assigned weights so that their probabilities are proportional to their relative frequencies. The estimator is proved to be rank consistent. The theoretical property of the model is substantiated by empirical results. The new estimator outperforms the DOP1 estimator on the OVIS corpus.